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Overview 



This report presents findings from the Massachusetts Family Child Care study, a two-year 
evaluation designed to examine the impacts on providers and children of an early childhood 
education program aimed at improving the development and learning opportunities in the care 
settings and, as a consequence, the outcomes for children in care. The early childhood education 
program— focuses on training caregivers to stimulate children’s cognitive, 
language, and social-emotional development through game-like interactions with individual 
children across the day. This evaluation of LeamingGames is one of four state experiments that 
were conducted as part of the Evaluation of Child Care Subsidy Strategies, whose overall 
ohjective is to determine how differences in certain aspects of child care subsidy policies and 
quality improvement initiatives are related to outcomes for parents, children, and/or child care 
providers. The study is being conducted by Abt Associates Inc, with its research partners MDRC 
and the National Center for Children in Poverty of Columbia University, under a contract with the 
Administration for Children and Families within the U.S. Department of Flealth and Human 
Services. 

The Massachusetts Family Child Care Study tested the value of an effort to improve the quality 
and outcomes of family child care, as many low-income families who receive subsidies choose 
this form of care for their young children. As such, the study addresses important policy 
questions for Massachusetts and for other states about how to enhance the skills of the early 
education workforce to improve the quality of children’s experience in child care settings. Since 
home-based providers typically care for a small number of children, family child care appeared to 
be an environment that could support the development of these individualized, responsive 
relationships between the provider and the children in care. 

LeamingGames was designed to promote children’s cognitive and language development through 
learning opportunities provided by their caregivers. LeamingGames focuses in particular on 
increasing the frequency of rich language interactions between caregivers and individual children. 
This emphasis grows out of the evidence of the importance of oral language development in 
children’s understanding of words and concepts, in their ability to become competent readers, and 
in their long-term academic success and of the role that rich language stimulation plays in 
promoting children’s development. This evaluation of LeamingGames examines the 
effectiveness of the program in increasing the frequency of rich, stimulating, individualized 
interactions as a pathway to improved developmental outcomes for the children who are cared for 
by providers trained on LeamingGames. 

The study addressed two major research questions: 

• Did LeamingGames have significant positive impacts on the developmental support 
provided by providers to the children in care? 



LeamingGames is a series of early learning activities developed for the Abecedarian Project. 
MindNurture, a subsidiary of Teaching Strategies, Inc. created and currently disseminates the 
curriculum (http://mindnurture.com). 
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• Did LeamingGames homes have significant positive impacts on developmental 
outcomes for the children in care? 

The study was conducted with a sample of family child care providers that are members of family 
child care networks in Massachusetts. There were 55 child care networks in the state at the time 
of the study, and 18 of the 55 met study eligibility criteria of having sufficient home visiting staff 
to implement the design and having in place a system of general technical assistance for the 
family child care providers in the network. Fifteen of the 1 8 networks agreed to participate in the 
study. Within each network, all eligible family child care providers were recruited, where 
eligibility was defined as having at least one child less than 36 months of age in care. 

In each of the family child care networks, half of the providers who agreed to participate in the 
study were randomly assigned to LeamingGames and half were assigned to the control condition. 
All of the networks who participated in the study indicated that they already provided or would 
provide two home visits per month to family child care homes. During these visits, the 
LeamingGames providers were trained on LeamingGames. Network administrators selected the 
home visitors who provided LeamingGames support from among their staff. The LeamingGames 
home visitors did not work with any of the providers in their network who had been assigned to 
the control group. The home visitors who were chosen to support the LeamingGames providers 
were trained on LeamingGames by the developer. This involved an initial three-day training, 
with quarterly one-day follow-up trainings and additional technical assistance and support. 

The LearningGames Approach 

LeamingGames is designed to train caregivers to support children’ s development through 
structured, game-like, one-on-one interactions that promote the child’s acquisition of oral 
language (vocabulary, concepts) and socio-emotional development. LeamingGames also 
encourages caregivers to find opportunities throughout the day to enhance children’s 
development, by engaging in positive, individualized verbal interactions in the context of play 
and physical care, as well as in more structured activities such as reading aloud. The 
LeamingGames approach includes four primary components: 

• LeamingGames activities. 200 games, organized by child age, for providers to use 
with children from birth to 5 years, along with written descriptions of the games, 
ways to implement them and parent handouts (English and Spanish) that describe 
each game. 

• Enriched caregiving. Guidance for providers on how to implement cognitive and 
language stimulation across all activities during the day, especially during routine 
care and ordinary activities that may not typically be utilized as learning 
opportunities for children. 

• Conversation books. Small, informal booklets (4-6 pages), which focus on early 
language concepts, for the provider to use for one-on-one interactive reading. 

• Guidance on cognitive developmental sequences. Specific instructional sequences 
for providers to use with children at different stages of development, to help children 
acquire more sophisticated cognitive skills, in particular, the “3S system” (See, 

Show, Say) and the “3N system” (Notice, Nudge, Narrate). 
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In addition, the LeamingGames approach includes forms for providers to document weekly 
planning for each child in care, including the games planned and actually used with each child 
during the week. 

LeamingGames was adapted from a parent curriculum that was first used in the Abecedarian 
Study, a two-generational intervention in which parents and infants together attended an intensive 
education program at a center. Infants received high-quality group care with trained teachers, and 
parents received parenting education and worked with their own children in cognitive-stimulating 
activities, with coaching and support from program staff. The Ahecedarian Program had 
substantial short-term impacts on infants (Ramey et. ah, 2000; Ramey & Campbell, 1984; 
Campbell & Ramey, 1994) and, in 20-year follow-up studies, has continued to show significant 
impacts on the children and their mothers (Campbell, Pungello, Miller-Johnson, Burchinal, & 
Ramey, 2001). While LeamingGames and its precursor have been studied in center-based care, 
home visiting programs and parenting programs, it has not been studied previously in family 
child care homes. 

Implementation of LearningGames 

The implementation of the LeamingGames evaluation involved a complex set of organizational 
relationships among the state Office of Child Care Services^, child care resource and referral 
(CCR&R) networks, the family child care networks and their providers, and the developer and 
trainers of LeamingGames. The implementation of the intervention relied primarily on a “train 
the trainer” approach. Selected home visiting staff from the participating family child care 
networks were trained by Dr. Joseph Sparling, the developer of LeamingGames. Two half-time 
project coordinators from the CCR&Rs received funding to provide the network home visitors 
with ongoing support and technical assistance and attempted to maintain monthly contact with 
them. In the second year of the study. Dr. Sparling and his staff also offered additional direct 
training on LeamingGames to participating family child care homes. In turn, home visiting staff 
were asked to provide training and support to LeamingGames during two visits each month. 

Working with home visitors trained in the approach, LeamingGames providers were expected to 
use specific games with individual children, eventually incorporating and deepening each game 
as time progressed. In addition, they were asked to enrich caregiving routines using the 
LeamingGames learning techniques. A home in which LeamingGames is fully implemented is a 
language-rich environment in which a provider spends a substantial proportion of her time 
focusing on and interacting with one or two children at a time, and opportunities for children’s 
learning and development are provided throughout the day. 



^ Early in the study the state reorganized its child care and early childhood programs and created the 
Department of Early Education and Care. 
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Study Design 



The study was a randomized cluster design, with family child care providers assigned to 
treatment (LearningGames) or control (husiness-as-usual) within networks, and children 
clustered within provider, so that all children in treatment homes had an opportunity to receive 
LeamingGames and all children in control homes received the providers’ regular program. The 
study is intended to he an effectiveness study, in that the impact of LeamingGames was studied 
under typical or real-world conditions. Although the study tried its best to achieve high fidelity 
of implementation across all treatment providers through training and ongoing support, variation 
in implementation was expected. Since the study was being conducted under real world 
conditions, providers had control over whether and how well they implemented the intervention 
procedures. 

Recruitment and Random Assignment 

As stated earlier, 1 8 of the 55 family child care networks that had contracts with the state of 
Massachusetts were recruited to participate in the study. Of these, 15 ultimately agreed to do so. 
They, in turn, helped recruit 353 family child care homes to be part of the evaluation. The study 
used within-agency random assignment so that approximately half of the homes from each 
network were assigned to the LeamingGames group and half to the control condition. The 
process resulted in 173 homes being assigned to LeamingGames and 180 assigned to the 
business-as-usual control group. 

Sampie Attrition 

The sample suffered from substantial attrition of providers between baseline and the posttest, two 
years after the intervention began. Attrition in the provider sample derives from two sources: 
individual providers who dropped out of the study (provider-level attrition) and entire agencies 
that dropped out (agency-level attrition). Since random assignment was conducted within 
agency, agency attrition resulted in the loss of approximately equal numbers of treatment and 
control providers. Consequently, loss of an agency does not bias the sample but does reduce the 
power of the analyses to detect impacts. Provider-level attrition, on the other hand, both reduces 
power and potentially introduces bias into the sample. 

At the end of the two years of implementation of the LeamingGames, four networks had dropped 
from the study and the overall attrition rate was 58%. There was higher attrition among the 
LeamingGames providers, compared with the control providers (60% versus 55%). About half 
of the attrition from the provider sample was the result of the four agencies dropping out (108 
providers). The remaining attrition was the result of 95 providers who withdrew for individual 
reasons. The provider-level attrition varied widely across the 18 agencies remaining in the sample 
after two years. Four of the agencies lost at least half of their providers, while other agencies lost 
less than 10% of their sample. 

The study was designed to assess provider and child outcomes after providers had been 
implementing LeamingGames for two years. As shown in Exhibit E.l, the level of provider 
attrition after two years was 58%, which translates into 150 providers remaining in the study. All 
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of these providers were part of the observations of provider behavior that were conducted in fall 
2007. This level of attrition required adjustment for baseline differences between the samples of 
LeamingGames and control providers in the analyses. 



Exhibit E.1 : Provider Sampie and Attrition Over the Study 





30 Months Post-Random Assignment 
(2 Years of Implementation) 


T 

(n = 173) 


C 

(n = 180) 


Total 
(n = 353) 


Providers Remaining 


69 


81 


150 


Overail Attrition 


60.1% 


55.0% 


57.5% 


Sources of Attrition 








Agency Attrition® 


28.9% 


32.2% 


30.6% 


Individual Provider Attrition 


31 .2% 


22.8% 


26.9% 



At the end of two years of LeamingGames, the study design called for assessment of outcomes 
for children in the homes who were between 12 and 60 months. Child assessments were only 
conducted in homes where there was at least one child enrolled who met three criteria: age (at 
least 12 months of age and not yet in kindergarten), time in care (at least 6 months in care with 
provider), and parent permission. In addition, the provider herself had to agree to continue to be 
part of the study data collection. Child assessments were not conducted in 29 of the 150 provider 
homes who were observed after two years of implementation. The remaining sample of 121 
providers reflects a very high level of cluster attrition, which introduces a potentially high level of 
bias into the child impact analyses. Further, at the time of the assessments, although 374 children 
in the sample of participating homes met the study criteria for age and time in care, only 9% of 
these children had been in the homes for two years, since the time of random assignment. The 
impact analyses were unable to adequately control for baseline differences in the LeamingGames 
and control children who were assessed, since more than 90% of these children were not in care 
at baseline. 

This low proportion of children in the homes at baseline and two years later is not unexpected. 
First, any children in the homes in spring when the study began who were at least 3 years of age 
would have entered kindergarten by the time of the assessments and would therefore not be 
eligible to be tested. Second, over the more than two-year span from baseline to the child 
assessments, many children would have moved out of the original homes. Even for the 9% of the 
children who were present in the homes at random assignment and for the assessments, the study 
does not have child-level baseline data to examine differences between treatment and control 
children at baseline or to adjust for any differences that do exist.^ Therefore, the study team 



Home visitors were asked to assess children in the homes at baseline using Ages and Stages. By the 
end of summer 2005, a small percentage of assessments had been done. It was clear that requiring 
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concluded that the child assessment sample cannot he assumed to support credible analyses of the 
impact of LeamingGames on children. Before reaching that conclusion, impact analyses were 
conducted on the children who were assessed. In the interest of transparency, we have presented 
the results of these analyses in Appendix B of the full report 

Data Collection 

Data obtained for the study came from multiple sources. To address questions on implementation 
information came from interviews with providers, home visitors, network administrators, and 
LeamingGames trainers; review of tracking documents of technical assistance activities; and 
ratings of the fidelity of implementation of LeamingGames conducted by home visitors as well as 
study staff. To address the research question on provider impacts, provider outcomes were 
measured through direct observation by the agency home visitors and by independent study staff. 
Observations of the homes were conducted at three times over the intervention period: at 
baseline, after one year of implementation of LeamingGames, and after two years of 
implementation. The impact analyses focused on the observations undertaken at the end of the 
intervention. At this time point, all treatment group providers remaining in the sample had been 
in the study for more than two years, and treatment providers had received at least 24 months of 
systematic training support on LeamingGames from the network home visitors. (The range of 
exposure varies somewhat, depending on the month providers received their initial training.) 

The observation measures used for the data collection included: 

• The QUEST Caregiver Rating Scale , which assesses the behavior of the adult who is 
caring for the children in six areas: caring and responding, supporting social-emotional 
development, supporting play, supporting cognitive development, supporting language 
development and early literacy, and television and computers. 

• The Snapshot of Activities (OMLIT-SNAP) , which is a time-sampled description of child 
activities and groupings, integration of literacy in other activities, and language in the 
setting. It has two sections. The Environment section describes the number of children 
and adults present, as well as the type of adult (staff, parents). The Activities section 
describes activities that are taking place. Then, for each activity, the observer records the 
number of children and adults in that activity, whether any adult or child is talking, 
whether they are speaking English or another language, whether any literacy materials 
are used (text, writing, letters, and if there is singing with the children (distinguished on 
the measure because of its potential as a phonological awareness/oral language support). 

• The Read Aloud Profile (OMEIT-RAP) is a description of adult behavior when reading 
aloud to children. The RAP records adult behavior during the read-aloud session on 
supports for comprehension, questions, attention to print knowledge, and vocabulary. 

• The TAEK , developed for the LeamingGames study, assesses the extent to which 
providers engage in extended conversation with individual children, as well as other 



home visitors to complete children’s baseline assessments would further delay the implementation of 
the intervention so this requirement was dropped. 
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types of one-on-one language interactions that could build children’s oral language skills. 
The TALK codes provider/child verbal interaction in five categories: management or 
helping, provider only (including provider narrating child’s actions), simultaneous 
verbalization (singing, chanting, rhymes), discussion (short, fewer than 4 turns back and 
forth), and extended discussion (4 or more back and forth turns, with provider building on 
child’s responses). 

• The Arnett Caregiver Interaction Scale (CIS) is a 26-item rating scale that assesses the 
quality and content of the caregiver’s interactions with children. The scale was designed 
to provide information on various socialization practices that have been identified in 
research on parenting. The scale can be used without modification in both center and 
home-based settings. The items measure the emotional tone, discipline style, and 
responsiveness of the caregiver in the setting. 

Findings: Implementation of LearningGames 

The Professional Development Model 

The study did not formally measure the fidelity of the professional development model. Evidence 
on fidelity comes from stakeholder interviews and a review of notes and logs. Together, these 
sources indicate that the professional development model was partially implemented, in terms of 
the fidelity of the home visiting protocol, the adherence to twice monthly home visits, and the 
provision of technical assistance by project coordinators. 

The major barriers to implementation fell into three categories, discussed below. 

Roles and Responsibilities for the LearningGames Implementation 

Even initially, the lines of authority for LearningGames implementation were complicated and 
not always clear. They became more complicated when, in the first year of the implementation of 
the intervention, the Massachusetts Department of Early Education and Care (EEC) decided that 
while it would endorse the study and provide funding for the intervention, it should not be 
directly involved in the implementation of the LearningGames professional development model. 

Train-the-trainer Approach Using Home Visiting Staff 

The study did not anticipate the amount of time that would actually be needed for home visitors 
to master the LearningGames approach. Some of the home visitors who had relatively low levels 
of education and child development experience seemed to require even more technical assistance 
and support than was originally anticipated. Providing home visitors with adequate time for 
mastery of the approach was further impeded by high home visitor caseloads and a high level of 
turnover among home visitors. High caseloads also appeared to impede some home visitors in 
consistently conducting the twice monthly home visits called for by the professional development 
model. 

Support By Project Coordinators and MindNurture Staff 

Since project coordinators received training at the same time as did the home visiting staff, they 
did not have the opportunity to master fully the approach before they were asked to support its 
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use by the home visiting staff. In addition, in the first year of the study, while Dr. Sparling and his 
staff provided more training than was originally planned, MindNurture did not receive the level 
of resources necessary for staff to be in the state providing direct support to the project 
coordinators and networks to the degree that likely was needed, in large part because these needs 
were not well understood prior to implementation. More funding added in Year 2 facilitated his 
ability to provide more technical assistance to staff as well as direct training to providers. 

Providers’ Implementation of the LearningGames Program Model 

Even though there were barriers to the implementation of the LearningGames professional 
development model, it appears that LearningGames providers were using the approach at least to 
some extent. Abt Associates created an 1 1 -item fidelity rating scale using those items from the 
provider observation that were most closely aligned with the LearningGames approach. The 
fidelity measures items included practices that could happen in any home; therefore it is possible 
for the comparison group also to be practicing these behaviors as they are consistent with high- 
quality caregiver-child interactions. For each of the items on the fidelity measure, a three-point 
scale was developed, with a rating of “3” indicating that providers were engaging in the activity 
at a level that would be considered “fully implementing” the LearningGames approach. The total 
number of possible points on the fidelity scale ranged from 1 1 to 33. 

There was a significant difference on the fidelity score between the LearningGames providers 
and the control providers (Exhibit E.2). The average fidelity score for the LearningGames 
providers was 18.4, compared with an average score of 15.3 for the control providers. In 
addition, we created a score based on the proportion of items on which a provider received a 
rating of “1,” meaning that they never or infrequently exhibited the behavior; therefore receiving 
the lowest rating on multiple items was seen as an indicator of low fidelity to the LearningGames 
approach. The average proportion of items with a rating of “1” was 50% for the LearningGames 
providers and 69% for the control providers, which was significantly different. This suggests that 
the LearningGames training was effective at changing the behavior of the family child care 
providers in line with the LearningGames objectives. 



Exhibit E.2. Scores on Fidelity Scale after Two Years of LearningGames Intervention by 
Treatment Status 


Measure 


Treatment 

Providers 


Control Providers 


Statistical 
Significance of 
Difference 


Mean 


Mean 


p- value 


Fidelity rating (out of 33) 


18.4 


15.3 


.03 


Proportion of fidelity items 
where provider scored as “1” 


50% 


69% 


.0001 
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Findings: Provider Outcomes 



Outcomes after One Year of Implementation 

As stated earlier, the sample suffered from significant attrition. Therefore, the first question tested 
was whether the samples of treatment and control providers who remained in the sample after one 
year were statistically different from each other, either on demographic characteristics or 
interactions with children. For the observation measures administered at haseline, the 
LearningGames and control providers in the one-year observation sample did not differ at a 
statistically significant level on the QUEST Caregiver Rating Scale and the Arnett CIS, which 
were administered at baseline. In addition, the LearningGames and control providers in the one- 
year observation sample did not differ at a statistically significant level on key demographic 
characteristics. 

The analyses of the one-year provider outcomes were considered exploratory, since the study 
design called for the primary test of impacts on providers to be based on outcomes at the end of 
the full two years of the intervention. Differences in the behavior of the treatment and control 
providers were evaluated based on five observation measures that were administered by 
independent observers at the end of one year of LearningGames. 

From instruments administered after one year of implementation, 18 outcomes were constructed 
to assess the impact of LearningGames on family child care providers’ instructional approaches 
and caregiving activities. Some of the outcomes were designed to assess instructional strategies 
that were specifically targeted by LearningGames, particularly those associated with enriched 
caregiving and interactive book reading. Other outcomes represented high-quality practices to 
support children’s learning and development. Together, the outcomes provided a broad portrait 
of the types of activities, interactions, and instructional approaches that providers use in family 
child care homes. 

There was a statistically significant difference between the LearningGames and control providers 
on only one of the 18 provider outcomes tested: proportion of time that providers used “enriched 
caregiving” (Exhibit E.3). This outcome described the provider’s introduction of cognitively- 
stimulating language play and interactions during caregiving routines such as washing hands and 
eating snack; narrating, talking about, giving feedback on, or asking questions about what 
children are doing; and nudging children to try something new or to extend an activity by 
themselves. It should be noted that testing this number of outcomes means that by chance alone, 
one or two of the contrasts may be significant. 
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Exhibit E.3 Impacts on Provider Practices after One Year of LearningGames Intervention 








Statistical 




Treatment 


Control 


Significance 

of 




Providers 


Providers 


Difference 




% 


% 


p-value 


OMLIT Snapshot of Activities 








Children involved in high-value activities® 


50.2 


53.2 


.15 


Provider highly Involved In children’s activities (Instructing, 
reading, discussien) 


31.0 


32.3 


.65 


Provider not involved In children’s activities 


33.0 


35.0 


.39 


Provider Interactions with Children 








Enriched caregiving with one or two children 


35.9 


28.8 


.04* 


QUEST Caregiver Rating Scale (1- 3) 


Mean 


Mean 




Provider support for cognitive, language, and social development 


2.09 


2.12 


.57 


Arnett Caregiver Interaction Scale (1- 4) 


Mean 


Mean 




Responsive 


3.36 


3.28 


.35 


Warm 


3.76 


3.70 


.41 


Attached/Engaged 


3.63 


3.57 


.41 


Permissive 


3.05 


3.03 


.84 


OMLIT Read Aloud Profile 


% 


% 




Reads aloud to one or two children 


34.7 


32.4 


.73 


Uses “see, show, say with one or two children 


31.6 


29.5 


.74 


Points out features of print 


90.8 


87.6 


.46 


Points out sounds/letters or sound-letter link 


7.1 


12.4 


.21 


Prometes print motivation 


75.5 


77.1 


.78 


Introduces/highlights vocabulary 


25.5 


25.7 


.97 


Supports comprehension: provides Infcrmation 


90.8 


85.7 


.26 


Supports comprehension: links to children’s experience 


31.6 


27.6 


.53 


Supports higher order thinking through the use of questions 


12.2 


12.4 


.98 


Sample 


98 


105 




1 “Includes reading and literacy activities; dramatic, creative, sensory and fine motor play, blocks, and games I 


Key: * = p < .05; ** = p < .01 









Outcomes after Two Years of Implementation 

At the time of the two year observations, 150 providers from the originally-assigned sample of 
353 providers remained in the study, 69 LearningGames providers and 81 control providers. 
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There were no significant differences between the LeamingGames and the control providers in 
the analytic sample on the QUEST Caregiver Rating Scale and the Arnett CIS. Also, as with the 
Year 1 sample, for selected demographic variables, the LeamingGames and control providers 
remaining in the analytic sample were not statistically different. 

Three over-arching constructs were developed from the observation measures as two-year 
outcomes for the analysis of impacts of LeamingGames on providers. The constructs were 
developed to align closely with the objectives of the LeamingGames program. Although the set 
of observation measures that were administered at posttest provided a rich set of data that could 
be used to assess impacts on providers, the analyses used a small number of constructs to avoid 
problems associated with multiple comparisons. That is, when a study examines many outcomes 
or findings simultaneously, the statistical significance of findings may be overstated. Without 
accounting for these multiple comparisons, the likelihood of finding a statistically significant 
finding increases with the number of comparisons. 

For the two-year impact study, our objective was to create three reliable constructs that, based on 
the items that made up each construct, measured outcomes that were aligned with the outcome 
goals of the intervention.. Specifically we developed constructs that measured the following: 

• The amount of time the provider was engaged with individual or pairs of children in 
extended language interactions with cognitively -rich content to assess the provider’s 
engagement in LeamingGames activities (or LearningGames-like activities); 

• The provider’ s availability to, positive interactions with, and responsiveness to children, 
across all activity contexts to assess the provider’s responsiveness to children; and 

• The extent to which the provider supported children’s oral language comprehension, 
across all activity contexts to assess the provider’ s support for learning vocabulary and 
concepts, as in the interactive book reading. 

In Year 2, LeamingGames had statistically significant impacts on all three provider outcomes. 
That is, compared with the control providers, the LeamingGames providers had substantially 
higher frequencies of rich oral language interactions and of interactions presumed to support 
children’s understanding of vocabulary or concepts, and they had significantly higher ratings on 
their responsiveness to the children. The effect sizes for the treatment-control differences were 
nearly half a standard deviation, which by convention is labeled a moderate effect size. Also, 
none of the provider baseline covariates was a statistically significant predictor of the three 
provider outcomes. The consistency of the findings suggests that the LeamingGames 
intervention, despite the apparent variability in the extent to which providers implemented a fully- 
realized model, was able to make a significant difference in how providers talked to and 
interacted with the children in their care. (See Exhibit E.4.) 
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Exhibit E.4: impacts of LearningGames on Provider Behavior 





Control 

Group 

Mean 


Treatment 

Group 

Mean 


Treatment 

Effect 


SE 


Statistical 
Significance 
of Impact (t- 
value) 


Rich oral language 
interactions 


-0.18 


0.22 


0.40** 


0.11 


3.72 


Support for development of 
vocabulary /comprehension 


-0.17 


0.20 


0.37** 


0.10 


3.58 


Responsiveness to children 


-0.19 


0.23 


0.47** 


0.16 


2.97 


Key:* = p < 0.05; ** = p < 0.01 



Conclusion 

We believe that further study of LearningGames in family child care is merited for two major 
reasons: (1) the study was able to show impacts on providers despite substantial variation in 
implementation of the professional development model; and (2) the study was not able to provide 
evidence of whether or not the changes in providers led to meaningful improvements in child 
outcomes. Both of these points are discussed below. 

This study showed that LearningGames had statistically significant positive impacts on the 
behavior of the family child care providers who received up to two years of support in 
implementing the program. LearningGames was effective at promoting high-quality, 
individualized and small group interactions between providers and children, which have been 
shown in previous research to be associated with children’s cognitive and language development. 
The effect sizes of these outcomes ranged from .37 to .47 standard deviations, which the field 
considers to be of moderate size. The study identified many barriers to implementation of the 
professional development model that could be addressed in future implementations. With a 
stronger implementation model, it is possible that the impacts on providers could have been larger 
than occurred in the Massachusetts implementation. 

Further, the study could not provide credible evidence about the impacts of LearningGames on 
children, since the high level of attrition in the child sample and the lack of a baseline assessment 
to verify that observable characteristics across the treatment and control group children assessed 
did not differ posed insurmountable threats to the internal validity of the estimates. This was 
compounded by threats to external validity, since there was no way to compare the assessed 
sample to the original one. 
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